fix(e2e-tests): increase timeouts for AWS tests MONGOSH-1924 #2286
Conversation
This should hopefully address AWS auth test flakiness on s390x, where DNS queries can take multiple seconds each due to the host CI setup.
nirinchev
left a comment
nirinchev
left a comment
There was a problem hiding this comment.
Is there something we can work on together with devprod to reduce DNS lookup times on these hosts? If you've already talked to them, might be good to add a comment in the code that summarizes the motivation/intricacies of this setup, as well as any steps we might want to take in the future. To be clear, this seems like a reasonable bandaid, but ideally, we'd want to improve the latency to avoid wasting CI resources.
|
@nirinchev Oh yeah, I was chatting with Thomas from the DevProd team who also helped with providing debug access to a host. I don't think there's a known root cause here – but they're eager to investigate this on their side. |
|
Cool, if they have a ticket tracking this, mind if you link to it in a code comment as it'll provide valuable context for future readers. |
|
@nirinchev I don't think there's a ticket right now. We do have:
so that's likely going to help the DevProd team with keeping an eye on things, and if we feel that this is really an issue that is significant enough to require special attention, we can always open a new ticket with them. But I also have to be honest and say that this problem doesn't seem bad enough that it requires extra attention at this point – if the tests start timing out with the new 1 minute limit I'd be a lot more concerned. |
|
@addaleax fair enough - I agree it's sufficiently unimportant that creating a ticket is not something we need to do at this point. |
This should hopefully address AWS auth test flakiness on s390x, where DNS queries can take multiple seconds each due to the host CI setup.
Example strace that indicates a 5 second delay for a DNS query: https://docs.google.com/document/d/1jWxKTZWma5nN5I3a9iTANnulzm_9GlLanEUbf9q9vh0/edit?tab=t.0